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The Honorable City Council 
City of Los Angeles 

City Hall, Room 395 

Los Angeles, California 90012 


Attention: Rules, Elections, and Intergovernmental Relations Committee 


Dear Honorable Members: 


SUBJECT: PERFORMANCE AUDIT AND REVIEW OF THE DEPARTMENT OF CANNABIS 
REGULATION’S PHASE III ROUND 1 LICENSING PROCESS 


SUMMARY 


The Department of Cannabis Regulation (Department) has received the “Department of Cannabis 
Regulation's Phase |I| Round 1 Licensing Process” Audit (Audit) conducted by Sjoberg Evashenk 
Consulting, Inc. (Auditor) and accepts the findings and recommendations presented therein. 


Following a thorough review of the Phase 3 Retail Round 1 application process, the Auditor found that 
the Department conducted the process in good faith and found no evidence of bias or unfairness. While 
the Auditor found that there may have been confusion arising from imprecise messaging and that 14 
applicants gained early access to the licensing application, the Auditor concluded that the Department's 
normalization process effectively negated any benefits that those applicants received by their early 
access. 


Based on the Auditor's findings, the Office of the City Administrative Officer (CAO) recommended that 
the Department be “directed to complete the processing of Phase 3 Retail Round 1 processing and 
commence the necessary work required to conduct the Phase 3 Retail Round 2 process later this fiscal 
year.” The CAO further recommends that, “In conducting the next licensing process, Department should 
take into account the Auditor's recommendations relative to improving the process for future rounds of 
licensing applications.” 
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Although the Department agrees that its normalization process was reasonable, it acknowledges that the 
process and the policy itself can and should be improved in a number of substantive ways. 
Improvements should be based on the information obtained from the Audit, the Departments 
experiences in licensing and policy implementation thus far, and, most importantly, the feedback 
received from the local community and stakeholders that are impacted by the process. Based on these 
considerations, the Department seeks a series of legislative changes to provide practical strategies 
designed to establish a more equitable path forward in our cannabis licensing and social equity 
programs. As such, the Department respectfully recommends the following legislative amendments be 
considered before the commencement of Phase 3 Retail Round 2. 


RECOMMENDATIONS 
That the City Council, subject to the approval of the Mayor: 


1. Instruct the Department of Cannabis Regulation (Department) to report back to City Council on 
recommended legislative amendments to the definition of “Equity Share” based on best practices 
and input from Social Equity Program stakeholders; 


2. Request the Office of the City Attorney, with the assistance of the Department, to prepare and 
present an ordinance to amend Article 4 to Chapter X of the Los Angeles Municipal Code (LAMC) 
to: 


a. Allow Phase 3 Retail Round 1 Applicants who are eligible for further processing to receive 
Temporary Approval provided that the Applicant submits an Equity Plan, resubmits 
documents to demonstrate compliance with “Equity Share” (as to be amended), passes a pre- 
licensing inspection, receives authorization from the State, submits required attestations and 
indemnification agreements, and clears a background check through Live Scan for all 
Owners; 


b. Allow Social Equity Individual Applicants who submitted an application during the 14-day 
Phase 3 Retail Round 1 Application Window and were deemed ineligible for further 
processing, to receive priority processing for a new application for one commercial cannabis 
activity not subject to Undue Concentration findings (i.e., non-volatile manufacturing, 
distribution, or delivery) and provide them up to one year to submit a complete annual license 
application; 


c. Establish new eligibility criteria for the Social Equity Program for Phase 3 Retail Round 2 and 
all future Social Equity Program application processing, as follows": 


i. | Amend the definition of “Low Income” to mean low income as referenced in the annual 
U.S. Department of Housing and Urban Development (HUD) income limits? based 
upon the Area Median Income (AMI) for Los Angeles County, and which considers 
household size and assets; 


1 Individuals who have been previously verified as Tier 1 and/or Tier 2 Social Equity Applicants and wish to 
participate in the Phase 3 Delivery application processing will not have to reapply for verification and shall be 
required to own the requite percentage of Equity Share as required at the time of verification. 

2 The Department of Housing and Urban Development (HUD) sets income limits that determine eligibility for 
assisted housing programs. HUD develops income limits based on Median Family Income estimates and Fair 
Market Rent area definitions for each metropolitan area, parts of some metropolitan areas, and each non- 
metropolitan county. 


The Honorable City Council 
Performance Audit and Review Phase 3 Retail Round 1 
Page 3 


ii Amend the definition of “Disproportionately Impacted Area” from Zip Codes to Police 
Reporting Districts? as established in the Revised Social Equity Analysis with 10-years 
of residency to qualify; 


ii. | Amend the definition for “Social Equity Individual Applicant” to mean a natural person 
who meets any two out of the three following criteria, as amended: 1) Low Income; 2) 
Disproportionately Impacted Area; and, 3) Cannabis Arrest or Convictions; 


d. Require Social Equity Individual Applicants seeking to participate in Phase 3 Retail Round 2 
and all future Social Equity Program applications processing to: 


i Own at least 51 percent “Equity Share” (as to be amended) in the Social Equity 
Applicant seeking a License; 


ii, Qualify under the newly defined eligibility criteria; and, 


iii, Exempt from this provision the following Social Equity Individual Applicants seeking to 
participate in LAMC Sec. 104.06.1(f): 1) Delivery Pilot Program, 2) Phase 3 Retail 
Round 1 Social Equity Individual Applicants that received an ineligibility letter for 
further processing because its Business Premises was in a geographic area of Undue 
Concentration limits or within a 700-foot radius of another Type 10 application, and 
3) Phase 2 Applicants eligible for processing pursuant to Section 104.08, 


e. Amend the Phase 3 Retail Round 2 process to determine Social Equity Individual Applicants 
eligible for further processing-by lottery. The determination of eligibility to participate in the 
lottery will require applicants to be verified under the new definition of Social Equity Individual 
Applicant and clear a background check through Live Scan. Social Equity Individual 
Applicants who are deemed eligible will then be entered into a lottery to determine who is 
eligible to apply for the available retail commercial cannabis activity licenses; 


f Allow Social Equity Individual Applicants who have been determined to be eligible for further 
processing by lottery up to one year to submit a complete annual license application; 


g. Allow Applicants who are eligible for further processing to receive Temporary Approval 
provided that the Applicant submits an Equity Plan (if applicable), submits documents to 
demonstrate compliance with “Equity Share” (if applicable), passes a pre-licensing inspection, 
receives authorization from the State, submits required attestations and indemnification 
agreements, and clears a background check through Live Scan for all Owners; 


h. Amend the definition of “Equity Share” based on best practices and input from Social Equity 
Program stakeholders; and, 


i Require all Applicants and Licensees to submit an Equity Plan as established by the 
Department. 





3 Los Angeles Police Department reporting districts are currently published to the City’s GeoHub 
(https://tinyuri.com/ss3a7ms). If adopted, the Department will add a layer to its online mapping tool which will allow 
prospective Social Equity Individual Applicants to identify which reporting district is associated with an address. 
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BACKGROUND 


The Department has received the Audit and accepts the findings and recommendations 
presented therein. The purpose of the Audit (Attachment) was to conduct a performance 
review of the Department's Phase 3 Retail Round 1 application process to determine whether errors 
occurred on the morning of September 3, 2019, when licensing began and, if errors occurred, the 
nature of those errors and the degree to which they were material to the first-come, first-served 
application process. In October 2019, the Department paused the further processing of Phase 3 
Retail Round 1 applications at the Mayor’s request until the completion of the Audit. 


Following a thorough review of the Phase 3 Retail Round 1 application process, the Auditor found that 
the Department conducted the process in good faith and found no evidence of bias or unfairness. While 
the Audit found that there may have been confusion arising from imprecise messaging and that 14 
applicants gained early access to the licensing application, the Auditor concluded that the Department's 
normalization process effectively negated any benefits that those applicants received by their early 
access. 


Based on the Auditor's findings, the CAO recommended that the Department be “directed to 
complete the processing of Phase 3 Retail Round 1 processing and commence the necessary work 
required to conduct the Phase 3 Retail Round 2 process later this fiscal year.” The CAO further 
recommends that, “In conducting the next licensing process, Department should take into account the 
Auditors recommendations relative to improving the process for future rounds of licensing 
applications.” 


Although the Department agrees that its normalization process was reasonable, it acknowledges that the 
process and the policy itself can and should be improved in a number of substantive ways. 
Improvements should be based on the information obtained from the Audit, the Department's 
experiences in licensing and policy implementation thus far, and, most importantly, the feedback 
received from the local community and stakeholders that are impacted by the process. Thus, the 
Department seeks a series of legislative amendments, starting with those outlined in the 
recommendations section of this report, to improve our local licensing and social equity programs. 


As to Phase 3 Retail Round 1 specifically, the Department recommends that the City create an equitable 
path forward for all those that participated in the following two ways. First, the Department recommends 
that the City allow Phase 3 Retail Round 1 applicants eligible for further processing to proceed with 
further processing (as they have concluded initial processing and paid an invoice fee) and allow them to 
receive Temporary Approval, subject to conditions as established. Second, the Department recommends 
that the City create a new category of priority application processing for those Social Equity Individual 
Applicants‘, who participated in Phase 3 Retail Round 1 but were not determined to be eligible for further 
processing, to receive priority processing for one commercial cannabis activity not subject to Undue 
Concentration Findings (i.e., non-volatile manufacturing, distribution, or delivery), subject to conditions as 
established®. 


4 The term “Social Equity Individual Applicant” is used throughout this response to refer to a natural person who 
meets certain criteria proposed herein. This term is not to be confused with a “Social Equity Applicant,” which may 
be, but is frequently not a natural person, consisting of one or more Social Equity Individual Applicants and one or 
more other Persons that collectively submit an application for licensure. 

5 Local law also establishes limits for the number of cultivation and retail licenses that can be issued in any given 
Community Plan Area within the City. Once an area has reached its limit of permitted licenses, it is subject to a 
finding of Undue Concentration and the Department cannot accept applications located in that area unless an 
exemption is granted through the Public Convenience or Necessity (PCN) Process. Delivery, Manufacturing and 
Distribution activities are not subject to Undue Concentration findings. 
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For the Social Equity Program (SEP) generally, the Department recommends that the City strengthen the 
Program's provisions® to ensure that its purpose is fulfilled. The purpose of the SEP is to promote 
equitable ownership and employment opportunities in the cannabis industry to decrease the disparities in 
life outcomes for marginalized communities and to address the disproportionate impacts of the War on 
Drugs in those communities. In accordance with that purpose, the Department recommends that the City 
amend the SEP eligibility criteria’ to more appropriately target those individuals and communities. The 
Department recommends that the new proposed eligibility criteria apply to all SEP application processing 
in the future, except delivery application processing as already established. The Department further 
recommends clarifying and strengthening the definition of “Equity Share” to ensure that those most 
impacted have unconditional and direct control of their business. Furthermore, the Department 
recommends that all applicants participating in the SEP be required to comply with the new Equity Share 
requirements once finalized, including Phase 3 Retail Round 1 applicants. Lastly, to further the City's 
SEP goals, the Department recommends that the City require all commercial cannabis applicants and 
licensees to submit an Equity Plan as established by the Department. 


For Phase 3 Retail Round 2 specifically, the Department has two recommendations. First, the 
Department recommends that the application process be amended to allow Social Equity Individual 
Applicants to be determined eligible for further processing before they are required to secure a property 
and incur other business start-up costs. Second, the Department recommends that the application 
process be amended so that eligibility for further processing is determined by lottery, and not by an 
online, first-come, first-serve process. 


It is the Department's position that these recommendations provide practical strategies designed to 
establish a more equitable path forward in our cannabis licensing and social equity programs. As such, 
the Department respectfully requests that the City Council support the above recommendations before 
the commencement of Phase 3 Retail Round 2. 


The Department thanks the City Council for the opportunity to share our recommendations and looks 


forward to working with the City Council and stakeholders to continue to improve programming and our 
service to the City of Los Angeles. 


Sincerely, 


Ct fa 


CAT PACKER 
Executive Director 


CP:JK 


Attachment: PERFORMANCE AUDIT AND REVIEW OF THE DEPARTMENT OF CANNABIS 
REGULATION’S PHASE |I| ROUND 1 LICENSING PROCESS CAO REPORT 





£ Table 1 outlines the Original and Proposed Social Equity Program Definitions. 
7 Table 2 outlines the Original and Proposed Social Equity Program (SEP) Eligibility Criteria to be utilized by 
Application Processing Phase. 
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Table 1. ci and je Social “a tg Definitions 


Means 80 percent or below of Area Median 
Income for the City based on the 2016 American 
Community Survey and updated with each 
decennial census. 


Means low income = as 
referenced in the annual U.S. 
Department of Housing and 
Urban Development (HUD) 








Inclusive Option" as described on page 23 of the 


Low Income income limits based upon the 
Area Median Income (AMI) for 
Los Angeles County, and which 
considers household size and 
assets. 
Means an arrest or conviction in California for 
any crime under the laws of the State of 
California or the United States relating to the 
s f sale, possession, use, manufacture, or 
California cultivation of Cannabis that occurred prior to | No change 
Cannabis November 8, 2016. An arrest, prosecution or 
Arrest or conviction for a violation of Proposition D, as 
Conviction codified in former Article 5.1 of Chapter IV of the 
Los Angeles Municipal Code, is not a California 
Arrest or Cannabis Conviction. 
Means eligible zip codes based on the "More | Means Police Reporting 


Districts as established in the 











constitute the end of an original business; and 
(iii) the voting rights on fundamental decisions 
relating to the business. 





Disproportionately | "Cannabis Social Equity Analysis Report" | Revised Social Equity Analysis 
Impacted Area | commissioned by the City in 2017, and| with 10 or more years of 
(DIA) referenced in Regulation No. 13 of the Rules | cumulative non-continuous 
and Regulations. residency required to qualify. 
Means a share of all of the following: | Definition to be established 
(i) a business's profits, including dividends, | based on best practices and 
distributions or other payments; (ii) the proceeds | input from Social Equity 
of a sale of a business's assets, liquidation of a | Program stakeholders. 
= business, merger of a business into another 
Equity Share business, or another transaction that would 
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A Tier 1 Social Equity Applicant shall meet the | Define a “Social Equity 
following criteria at time of applying for a | Individual Applicant” to mean a 
License: 1. Low Income and prior California | natural person who meets any 
Cannabis Arrest or Conviction; 2. Low Income | two out of the three following 

Social Equity and a minimum of five years' cumulative | criteria, as amended: 

Individual residency in a DIA. 
Applicants e Low Income 


(Natural Person) 


A Tier 2 Social Equity Applicant shall meet the 
following criteria at time of applying for a 
License: 1. Low Income and a minimum of five 
years' cumulative residency in a DIA; or 2. a 
minimum of 10 years' cumulative residency in a 
DIA. 








Social Equity 
Applications 


(Entity Applying 
for a License) 








A Tier 1 Social Equity Applicant shall own no 
less than a 51 percent Equity Share in the 
Person to whom the License is issued 


A Tier 2 Social Equity Applicant shall own no 
less than a 33 1/3 percent Equity Share in the 
Person to whom the License issued. 


A Tier 3 Social Equity Applicant shall enter into 
a Social Equity Agreement with the City to 
provide capital, leased space, business, 
licensing and compliance assistance for a period 
of three years to Persons who meet the criteria 
to be a Tier 1 Social Equity Applicant or Tier 2 
Social Equity Applicant. 





* 10 or more years of 
cumulative —non-continuous 
residency in DIA(s) 

e Cannabis Arrest or 
Convictions. 

Social Equity Individual 


Applicant(s) shall own no less 
than a 51 percent Equity Share 
in the Person (Entity Applying 
for a License) to whom the 
License is issued 





Table 2. Proposed Social Equity Program (SEP) Eligibility Criteria by A) 


Phase 1 Existing Medical Marijuana Dispensaries (EMMDs) 


N/A 








Phase 2 Non-Retail Suppliers of EMMDs 


Original 





























Phase 3 Retail Round 1 Original 
Phase 3 Retail Round 2 Proposed 
Phase 3 Delivery — Pilot Original 
Phase 3 Delivery — General Proposed 
Manufacturing — General Proposed 
Distribution — General Proposed 








Public Convenience and Necessity (PCN) 


Criteria in effect when the 
Applicant's PCN request is 
approved by City Council 











ATTACHMENT 


FORM GEN 160 


CITY OF LOS ANGELES 
INTER-DEPARTMENTAL CORRESPONDENCE 


0220-057 16-0000 


Date: March 27, 2020 

To: The Mayor 

Cc: The Council 

From: Richard H. Llewellyn, Jr., City Administrative Officer 


Subject: PERFORMANCE AUDIT AND REVIEW OF THE DEPARTMENT OF CANNABIS 
REGULATION’S PHASE II] ROUND 1 LICENSING PROCESS 


SUMMARY 


In November 2019, this Office was directed by the Mayor's Office to utilize an independent, third- 
party auditor to conduct a performance audit and review of the Department of Cannabis 
Regulation's (DCR) Phase III Round 1 licensing process in response to concerns raised by 
applicants and other members of the public about the fairness of the Phase III Round 1 licensing 
process. This Office selected Sjoberg Evashenk Consulting (Auditor) from a list of auditors with 
existing contracts with the City, and the review of the DCR Phase III Round 1 licensing process 
commenced in January 2020. This report presents the final report from the Auditor (Attachment) 
for consideration. 


The Auditor found that errors in the DCR process allowed certain applicants early access to the 
application system. The DCR then created a “normalization process” to level the playing field 
and eliminate any advantage given to these early applicants. The Auditor determined that the 
DCR's “normalization process” was a reasonable way to address the errors and specifically was 
NOT making a recommendation that DCR use a different “normalization process.” Finally, the 
Auditor found that that the DCR conducted the Phase III Round 1 licensing process in good faith, 
and that there was no evidence of bias or unfairness. 


In light of the Auditor's findings that the existing “normalization process” created a fair and 
unbiased procurement process, this Office recommends that the DCR be directed to complete 
the processing of Phase III Round 1 license applications and commence the necessary work 
required to conduct the Phase III Round 2 licensing process later this fiscal year. In conducting 
the next licensing process, DCR should take into account the Auditor's recommendations 
relative to the improving the process for future rounds of licensing applications. 


Summary of Audit Findings 


As detailed in the Attachment, the following is a summary of the findings of the Auditor’s report. 
The Auditor found: 


1; 


No evidence that DCR's re-enabling of user accounts benefited any users over any other 
users and that the normalization process utilized by the DCR effectively negated any 
benefits for applicants that accessed the application before 10:00 AM the morning of 
September 3, 2019; 


There are two distinctive steps in the application process that should be identified: 1) 
signing-on to the Accela application software platform (Accela platform); and, 2) 
accessing the licensing application; 


A total of 14 applicants were able to access the licensing application prior to the 10:00 
AM opening time. This was due to two factors: 


a. Nine applicants had their accounts enabled prior to 10:00 AM due to manual 
password resets performed by DCR. Two of these applicants accessed the 
licensing application before 10:00 AM; 


b. 222 applicants signed-on to the Accela platform subseguent to the re-enabling of 
all accounts around 9:59:46. Twelve of these applicants were able to access the 
licensing application a few seconds prior to 10:00 AM; 


4. Two factors contributed to the 14 instances of early access to the licensing application: 


a. The method of disabling and re-enabling applicants’ accounts made the application 
process vulnerable to human error; 


b. Imprecise messaging by the DCR contributed to the misperception by some 
applicants that they had to wait until 10:00 AM to sign-on to the Accela platform 
rather than sign-on as early as possible to be able to access the application at 
10:00 AM; 


The normalization process utilized by the DCR using the time applicants accessed the 
licensing application was reasonable, appropriate, and effectively negated any benefit 
that any applicant gained from early access to the licensing application; 


An alternative normalization process based on sign-on times was identified based on 
information that became relevant in hindsight. The Auditor found that utilizing this 
alternative normalization process could potentially impact up to eleven applicants; 


The alternative normalization process identified by the Auditor was found to be no more 
reasonable or appropriate than the process utilized by DCR; 


“Se 


8. The Auditor recommends that the DCR: 


a. Develop a new method to restrict user's access to the application without 
deactivating accounts or resetting passwords; and, 


b. Make the Accela platform available to users well in advance of future application 
windows to avoid confusion between the application window and the sign-on 
process. 


Continuation of Phase III Round 1 and Commencement of Phase III Round 2 Licensing 


Following a thorough review of the DCR Phase III Round 1 licensing process, the Auditor found 
that the DCR conducted the process in good faith and found no evidence of bias or unfairness. 
While there were issues identified related to potential confusion resulting from imprecise 
messaging and the early access to the licensing application by 14 applicants, the Auditor found 
that the DCR’s normalization process effectively negated any benefits that applicants received 
from early access to the licensing application. Based on this information, the DCR should be 
directed to complete the processing of Phase III Round 1 license applications and commence 
the necessary work required to conduct the Phase III Round 2 licensing process and implement 
the Auditor's recommendations related to the application platform. 


RHL:NSC:02200142C 


Attachment: Phase III Round 1 Licensing Performance Review and Audit Report 


City of Los Angeles 
Office of the City Administrative Officer 
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REPORT HIGHLIGHTS 


RESULTS 
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BACKGROUND 


Created in 2017, the City 
of Los Angeles (City) 
Department of Cannabis 
Regulation (Department) is 
responsible for the 
administration of licensing 
for legal cannabis activity, 
including storefront retail, 
delivery retail, cultivation, 
manufacturing, distribution, 
and testing of commercial 
cannabis. 

The City established three 
phases to process 
commercial medicinal and 
adult-use cannabis license 
applications: 

Phase 1 - Existing 
Medical Marijuana 
Dispensaries (EMMDs) 

= Phase 2 - Suppliers of 
EMMDs involved in the 
cultivation, distribution, 
and/or manufacturing 
of cannabis products 

e Phase 3- General 
Licensing 

Applicants for Phase 3 
Retail Round 1 were pre- 
Verified as qualifying for 
the Department Social 
Equity Program. Once 
approved, applications 
would be accepted on a 
firstcome, first served 
basis beginning on 
September 3, 2019, 
at10:00 AM. 





KEY FINDINGS 


The Department and Accela, the application portal software vendor, closed access to the 
Accela system portal on August 28, 2019, by running a script to disable all user accounts in 
order to restrict potential access to the application prior to the start of the Application 
Window, To give Applicants access to the Phase 3 Retail Round 1 application by the 
Application Window start time of 10:00 AM on September 3, 2019, Accela ran a second script 
to re-enable all accounts around 9:59:46 Am. This allowed Applicants to sign on to their 
accounts and access the application; the Accela system recorded timestamps for these two 
events, account sign-on and application start. 

There are two distinct and separate steps that must be identified. 1) Signing onto the Accela 
platform; and 2) Starting the application process. There were 226 Applicants who accessed 
the platform, however, and more important to this inquiry, 14 Applicants that started the 
application process before 10:00 AM. The method of disabling and enabling accounts allowed 
the Applicants to access the application prior to the 10:00 Am Application Window, Two 
factors causing this are: 

o Manual password resets performed by the Department—between 7:42:40 Am and 
9:46:26 AM, passwords were reset manually and were not re-disabled. This was found 
to be unintentional, but resulted in two Applicants beginning their applications early—at 
9:50:12 AM and 9:56:03 AM. 


o After sign-on, no system barrier preventing Applicants from starting the application—In 
addition to the two Applicants described above, there were 12 Applicants that began the 
application process in the seconds before 10:00 av—between 9:59:46 AM and 9:59:59 
AW. There were 226 Applicants signed on to the Accela Platform prior to 10:00 AM. 


There is no evidence that any Applicant benefited over any other Applicant as a result of 
Accela's account re-enabling process. For those that started their application prior to 10:00 
AM, the Department implemented a reasonable “normalization” process that effectively 
negated any benefit the early application star. 

All accounts were enabled prior to 10:00 Am and many signed-on to the Accela system portal 
as early as possible; however, many Applicants waited until 10:00 AM to sign-on to the portal. 
In fact, 321 Applicants signed-on between 10:00:00 AM and 10:00:59 AM, 154 of which 
signed-on within the first two seconds of the hour. 


Some Applicants appeared to wait because of what, in hindsight, was imprecise messaging 
by the Department. In some cases, the Department stated that the Application Window 
would open at 10:00 Am, and in other cases the Department stated that Applicants would be 
unable to sign on to the Accela portal until 10:00 Am—the latter was inaccurate. 

e The potential misperception among some Applicants that they were prohibited from signing 
on to the Accela Civic Platform until 10:00 AM would have placed those Applicants at a 
disadvantage. 
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RECOMMENDATIONS 


This audit makes no recommendation regarding the outcome of the Phase 3 Retail Round 1 process. This audit found that 
while imprecise messaging and an ill-suited technique to eliminate access to the Phase 3 Retail Round 1 application prior to 
the Application Window led to some confusion regarding the application process, there is no evidence that the Department's 
process to re-enable user accounts benefited any users over any other users, and the Department's normalization process 
effectively negated any benefits for all 14 Applicants that accessed the application before 10:00 AM. Despite our conclusion 
that the resulting confusion lends legitimacy to an alternative normalization process, we found the Department's approach to 
normalization to be reasonable and appropriate, and we did not find alternative methods to be more reasonable or more 
appropriate than the adopted approach. 

In order to avoid similar confusion in the future, we recommend that, when establishing upcoming Application Windows, the 
Department of Cannabis Regulation develop a method that restricts all users from accessing the application without 
deactivating accounts or resetting passwords, and make the Accela Civic Platform available to all users well before the 
Application Window so as not to conflate the Application Window with the platform sign-on process. 
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Introduction and Background 


In March 2017, voters in the City of Los Angeles (City) passed Measure M, giving the City Council full 
authority to regulate cannabis-related activity within the City. Following the legalization of adult-use 
cannabis in the City, the City created the Cannabis Regulation Commission (Commission) and the 
Department of Cannabis Regulation (Department). Together, the Department and the Commission were to 
establish a regulated and transparent cannabis industry for the City. The Commission consists of five 
members appointed by the Mayor subject to City Council approval. The Commission's responsibilities are 
to provide oversight of the City’s regulation of the cannabis industry, monitor the City’s efforts relating to 
implementing and complying with cannabis laws and regulations, hold public hearings related to the 
issuance of commercial cannabis licenses within City limits, and make recommendations to the Mayor and 
City Council. The purpose of the Department is to administer licensing of legal cannabis activity, including 
storefront retail, delivery retail, cultivation, manufacturing, distribution, and testing of commercial cannabis, 
and to implement and administer the Social Equity Program. The Department consists of an Executive 
Director, two Assistant Executive Directors, a Social Equity Program Director, and 34 authorized full-time 
equivalent (FTE) positions. 


The Program Overview 


Commercial cannabis businesses must obtain a license from the State of California and from the City 
before engaging in commercial cannabis activity within the City. The type of license a business must obtain 
depends on the type of cannabis business it intends to operate. The City differentiates between Retail 
Activities, both storefront and delivery, and Supply Chain Activities, including cultivation, manufacturing, 
distribution, and testing. An annual license and temporary approval are two forms of licenses. 


The City of Los Angeles Municipal Code (LAMC) restricts commercial cannabis activity to certain locations 
based on zoning, sensitive use buffers, and undue concentration limits. Specifically, for storefront retail 
activity, undue concentration limits were determined by Community Plan Area and set at a ratio of one retail 
establishment per 10,000 residents. As such, it was determined that the City had the capacity to absorb 
approximately 250 new storefront retail businesses through the Phase 3 licensing process. In addition to 
these general restrictions, new storefront retail cannabis businesses may not be located within a 700-foot 
radius of any other storefront retail cannabis business or sensitive-use. 


The LAMC establishes a three-phase structure for the processing of license applications, beginning with 
the licensing of existing medical marijuana dispensaries (Phase 1), followed by existing suppliers of Phase 
1 businesses seeking licenses for non-retail activity including cultivation, manufacturing, and distribution 
(Phase 2), and culminating with the licensing of new commercial medicinal and adult-use cannabis 
businesses, including both storefront and non-storefront delivery retail (Phase 3). The City has already 
granted temporary approval to certain pre-existing cannabis businesses (i.e., medical marijuana 
dispensaries and associated supply chains) through the Phase 1 and Phase 2 licensing process, allowing 
them to operate pending the annual license application review and approval process. Ultimately, all 
temporary approval holders will be subject to an annual approval process. As of the end of 2019, both 
Phase 1 and Phase 2 have resulted in the temporary approval of 342 cannabis business licenses, with 
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another 362 Phase 2 applications pending. The Phase 3 application process, which is still in progress, 
consists of a total of five rounds; two rounds for new storefront retail business applications and three 
rounds for non-storefront delivery business applications. A total of 100 Applicants were to be determined 
eligible for further processing in Round 1, the focus of this report; and the remaining approximately 150 
Applicants were to be determined eligible for further processing in Round 2, which had been planned for 
Spring 2020 and has been postponed until after the completion of this audit. 


On April 30, 2019, the City Council instructed the Department to conduct Phase 3 Retail Round 1 in 
accordance with a specific timeline, which included performing a 60-day Applicant pre-vetting process for the 
City's Social Equity Program by May, 28, 2019 and starting to accept license applications by no later than 
September 3, 2019. The Application Window was to remain open for two weeks, concluding September 17, 
2019. Applications received during that time would be reviewed on a “first come, first served” basis, in the 
order in which the applications were submitted, with the first gualifying 75 Tier 1 Applicants and 25 Tier 2 
Applicants moving forward in the application process. 


Individuals seeking to participate in Phase 3 Retail Round 1 were reguired to meet minimum reguirements 
to be considered eligible for further processing. This included requiring Applicants to show proof of an 
existing lease, deed, or purchase agreement at a qualifying location, and to submit several forms, plans, 
and diagrams. As mentioned above, the Phase 3 Retail Round 1 application process also required 
Applicants to be pre-verified through the Social Equity Program, which is designed to give priority in 
application processing for new retail licenses and provide technical assistance and business development 
resources to Applicants representing communities that have been disproportionately impacted by the War 
on Drugs. This requirement distinguishes Phase 3 from Phase 1, which focused on, at least initially, issuing 
temporary approval to certain existing cannabis businesses operating within the City. 


The City required Applicants to be verified as qualifying for the Social Equity Program to be eligible to apply 
for processing in Phase 3 Retail Round 1. The Department accepted applications and verified the eligibility 
of Social Equity Applicants between May 28, 2019 and July 29, 2019. The Social Equity Program drew 
approximately 2,200 applications, of which approximately 1,600 Applicants were determined eligible to 
participate in the Phase 3 Retail Round 1 process. Individuals may qualify as a “Tier 1,” “Tier 2,” or “Tier 1 
& 2” Social Equity Applicant, based on a set of City-established criteria, summarized in Exhibit 1. 


EXHIBIT 1; SOCIAL Equity PROGRAM ELIGIBILITY REQUIREMENTS 

















SEP Tier Eligibility Requirements 
Ter’ Social Equity Applicants meet the following criteria 
1. Low Income and a prior California Cannabis Arrest or Conviction 
Ter2 Social Equity Applicants meet the following criteria. | 
1. Minimum 10 years’ residency ina Disproportionately Impacted Area 
Tier 1&2 Social Equity Applicants meet the following criteria. | 
4.__Low Income and a minimum five years’ cumulative residency in a Disproportionately Impacted Area 
For an application to be eligible for priority processing, the application must meet the following criteria. 
Application 4. Tieri priority processing, a Tier 1 or Tier 1 & 2 Social Equity Applicant must own a minimum 51 
Review percent equity share in the Applicant business entity 
2. Tier 2 priority processing, a Tier 2 or Tier 1 & 2 Social Equity Applicant must own a minimum 331/3 
percent equity share in the Applicant business 
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To support eligible applicants, the Department conducted several community trainings to prepare 
Applicants for navigating the Accela application platform and engaged in numerous community education 
programs, including city-wide workshops, approximately 250 one-on-one clinics, and online materials and 
training opportunities. 


On Tuesday, September 3, 2019, at approximately 10:00 Am, the Department opened the Phase 3 Retail 
Round 1 application process using the Accela platform. To complete an application, an Applicant was 
required to prepare and submit the following documents: 


e Executed lease agreement, purchase agreement, or property deed for business premises 
e Business premises diagram 
e Any and all Equity Share information to demonstrate compliance with the Social Equity Program 


e A series of standard forms created by the Department to facilitate the application review, including 
the Ownership and Financial Interest Holder Disclosure Form, Financial Information Form, 
Proposed Staffing and Security Plan Form, Labor Peace Agreement Attestation, Indemnification 
Agreement, and Radius Map Attestation 


Within the first hour of the Application Window, the Department received 656 applications. By the time the 
Department closed the application filing window on Tuesday, September 17, 2019, at 10:00 Am Pacific 
Daylight Time (PDT), the Department had received a total of 802 applications, inclusive of more than 40 
test and duplicate applications that were subsequently discarded. 


The first 100 applications that were submitted between September 3 and September 17 and found to 
comply with all requirements were considered eligible for further processing and invoiced. Department 
records show that all 100 invoiced Applicants paid their application fees on time. All other applications were 
determined to be ineligible because they were: not submitted within the top 100 qualifying applications; 
incomplete; a duplicate of a complete application previously submitted; and/or were for retail businesses 
located in an area with incompatible zoning restrictions, or that had reached undue concentration limits, 
and/or was within 700 feet of either an existing retail cannabis location, the planned location of a previously 
approved applicant, or a “sensitive-use” location, such as a school, daycare, or alcohol or drug treatment 
facility. 


The Application Process 


Prior to the opening of the Application Window, the Department provided extensive guidance and 
informational materials to the public regarding how the application process would unfold. In a thorough 
tutorial video, the Department described how Applicants were to access the Accela Civic Platform, sign-on 
through the Accela portal, navigate to the application screen, and complete the application. Specifically, the 
Department recommended Applicants first complete an “Application Checklist,” which informed Applicants 
of the information they would need access to during the sign-on and application process, such as the 
Applicant's username and password, the proposed business address and area of undue concentration, and 
other necessary information. 
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The Department informed Applicants that, on September 3, they would need to sign onto the Accela 
account using their user name and password. The Department repeatedly informed Applicants that they 
must ensure they are able to access their accounts prior to September 3 because, come September 3, if an 
Applicant experienced difficulties logging into their account because they did not have the correct user 
name and password, they would risk missing the Application Window. For instance, the Department notified 
Applicants that if they were blocked from accessing their account, 


*... the Applicant MUST visit the [Department] office in person to update their account to 
grant or restore access. Applicants will need to present a government-issued ID and Social 
Security Card to [Department] staff in order to verify ownership of the account. It is highly 
recommended that Applicants confirm account accessibility before September 3, 2019 to 
ensure that all documents and contact information in the system are correct. It is the 
Applicant's responsibility to secure their private access to the user account associated with 
their application in order to participate in the Phase 3 Retail Round 1 application process, 
which opens September 3, 2019.” 


As illustrated in Exhibit 2, once signed into their account, the Applicant is taken to their “Dashboard” page. 
This page allows the Applicant to navigate through different elements of their account, including the 
Applicant's Social Equity Application status, the Applicant's records, and other account information. The 
Applicant is then instructed to navigate to the “Home” screen, which gives Applicants the option to “Apply 
for a License” or “Search for a License.” Applicants were instructed to click “Apply for a License,” which 
would take the Applicant to a page that informs the Applicant of key terms and conditions, and requires the 
Applicant to read and accept the general disclaimer before beginning the application. Once accepting the 
terms and conditions, the Applicant is instructed to click the “Continue” button. This takes the user to the 
actual Application and represents the start of the application process. At this point, the Applicant is 
instructed to navigate through different elements of the application, including identifying whether the 
proposed business location is located in an area of undue concentration, entering the proposed business 
premises address, the types of commercial cannabis activity, upload the application documents described 
previously, and conduct a final review of the application information. Once complete, the Applicant would 
be notified that the Application is submitted and provided a unique Record Number to track its status. 


EXHIBIT 2. THE PHASE 1 RETAIL ROUND 3 APPLICATION PROCESS 
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The Licensing Platform 


In February 2017, the State of California selected Accela Government Systems (Accela, Inc.) to build the 
statewide cannabis licensing platform following its successful launch of a cannabis licensing platform in 
Denver, Colorado. In September 2017, the City piggybacked on the State's contract and entered into a sole 
source contract with Accela to develop and fully support the licensing and regulatory information system for 
cannabis activity in the City. Accela’s Civic Platform was developed with key components necessary to 
automate the intake and processing of online applications, fee collection and Applicant notifications 
throughout the various application and license stages. As it relates specifically to the Phase 3 Retail Round 
1 application process, the Accela system was designed with: 


e Specific timestamping functionality for critical user activities, including user sign-on, record initiation 
and submission, and document upload activities. The system was designed with server-side 
timestamped audit functionality and audit logging capabilities. 


e Record Identification Number (ID) generation to assign license Applicants a temporary application 
number as soon as the application started and a final Application Number when the application 
was submitted and recorded in the Accela database. For Phase 3 Retail Round 1, Accela and the 
Department established an Application Number that appears as “LA-C-19-310000-R-APP”, where 
“LA” represents the Accela's client account for the City of Los Angeles, “C” is assigned when itis a 
general application, “19" is the year the application was generated, “31” represents Phase 3 Retail 
Round 1, “0000” represents the order in which the Record ID was generated and corresponds to 
the order in which the applicaticns were started, and the “R” represents the activity (Retail 
Storefront). 


The Application Portal (Accela's Civic Platform) is used by individuals requesting Social Equity Program 
Applicant Eligibility Verification, applying for a license, and/or existing licensees who wish to maintain their 
license (i.e., renewal, update address, update to modify address, etc.). To access the Application Portal, 
users are reguired to first register a Public User account, which includes creating a username and 
password for use when signing into the system. 


To ensure prospective Applicants understood how to access and navigate the Application Portal during 
Phase 3 Retail Round 1, the Department issued numerous informational bulletins, alerts, videos, and 
brochures to the public and Applicants throughout the Social Equity Program pre-vetting process and 
application period. In the weeks leading up to the start of the Application Window, the Department 
instructed all eligible individuals to ensure they could sign onto their accounts prior to the opening of the 
Application Window and, ifthey could not sign on, they must have their accounts and passwords reset in 
person at the Department's office. Failure to do so would result in the individuals’ inability to access and 
participate in the Phase 3 Retail Round 1 application process. 


Consistent with advance noticing to potential Applicants, Accela ran a script that disabled all Phase 3 Retail 
Round 1 eligible user accounts in preparation of the launch of the Application Window and uploaded the 
final application workflow into the production environment on August 28, 2019. This process prevented all 
prospective Applicants from signing onto the Application Portal, while allowing Accela and Department staff 
continued access in the production environment of Accela’s Civic Platform as they prepared for the 
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Application Window and tested the application itself. In anticipation of the 10:00 Am opening of the 
application, Accela ran another script that re-enabled all user accounts around 9:59:46 Am. At that point in 
time, all prospective Applicants were able to sign onto the Application Portal, navigate to the application 
start page, and access the application—navigating to the application page typically took Applicants about 
32 seconds, though some navigated to the application page in just a few seconds. 


Concerns Over Early Access to the Phase 3 Retail Round 1 Application 


Once Accela had uploaded the final application workflow into Accela’s Civic Platform’s production 
environment oñ August 28, 2019, Department management continued end-user testing of the system to 
ensure it was working properly. This included submitting test applications. During this testing on the 
morning of September 3;:2019, the Department could observe the results of their submission of test 
applications on the backend, but also observed applications submitted by two individuals not associated 
with Accela or the Department. Both individuals started and submitted applications before Accela had re- 
enabled user accounts via script at 9:59 AM. The Department immediately began to research and determine 
who accessed the Civic Platform and the application before Accela had re-enabled user accounts and how 
they gained access. 


The Department quickly realized that the two individuals that began and submitted applications prior to the 
planned opening of the Application Window were among those who had their passwords manually reset by 
Department personnel the morning of September 3, 2019. Consistent with the Department's instructions, at 
least nine (9) individuals experiencing trouble signing onto the Civic Platform arrived in person at the 
Department's office to have their accounts reset. During this process, the Department manually re-enabled 
the user accounts so the potential Applicants could sign on using the Department-set temporary password, 
reset their personal password, and ensure they were able to sign onto their accounts prior to leaving the 
Department's offices. The Department, however, neglected to re-disable the accounts to ensure the 
individuals did not have access to the platform prior to Accela's mass re-enabling of accounts at 9:59:46 
AM. That is, while the manual resets were done in accordance with the Department's regular procedure for 
in-person password resets, the unique circumstances of that morning resulted in the unintentional granting 
of early access to those users. 


The Department subsequently informed the public of these two instances of early access and, in order to 
ensure these Applicants were not unfairly advantaged by starting the application early, decided in 
collaboration with Accela to “normalize” the application times to negate their advantage over other 
applicants. However, since this has come to light, the public has continued to raise concerns regarding the 
veracity of the outcome of the Phase 3 Retail Round 1 application process, as reported by the Department. 
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Scope and Methodology 


The City of Los Angeles, Office of the City Administrative Officer, (CAO) contracted with Sjoberg Evashenk 
Consulting, Inc., to conduct a performance audit of the Department of Cannabis Regulation's Phase 3 
Retail Round 1 Application Process. To ensure the confidence of the community in the City’s licensing 
process, the CAO sought an independent and objective third party to determine whether errors occurred on 
the morning of September 3, when licensing began, and, if errors occurred, the nature of those errors and 
the degree to which they were material to the first-come, first-served Applicant review process outcomes. 
Toward this end, the objectives of this audit were to: 


e Establish a comprehensive understanding as to whether any Applicants accessed the Accela 
platform prior to the Application Time and, if so, (1) how many Applicants, (2) which Applicants, 
and (3) the times at which they accessed the Accela platform; 


e Determine the root cause of any instances in which Applicants accessed the Accela platform prior 
to the Application time; 


e Assess any measures taken by the Department of Cannabis Regulation (Department) to address 
any instances in which Applicants accessed Accela prior to the Application Time; and 


e Assess how any Applicants were impacted, if at all, by any instances in which Applicants accessed 
Accela prior to the Application Time and any measures taken by the Department to address such 
instances. 


To accomplish the audit objectives, Sjoberg Evashenk Consulting held an Entrance Conference with the 
General Manager of the Los Angeles Department of Cannabis Regulation; conducted interviews with key 
Department management and staff, as well as representatives of Accela; reviewed relevant statutes, 
ordinances, rules, and regulations; reviewed general background information, including information made 
available through the Department's website, application materials, the Accela application portal, and 
internal guidelines and tools used by the Department to review Phase 3 Retail Round 1 applications; and 
obtained and considered complaints filed with the Department regarding the application process. 


In addition, we obtained and reviewed the Department's contract with Accela; an internal review of the 
Phase 3 Retail Round 1 application process conducted by Accela in October 2019; Accela Data System 
network performance reports, which reveal the load and latency of the server before, during, and after the 
application opening period of September 3, 2019, at 10:00 AM; and Accela Phase 3 Analysis and 
Workflows, which depicted the application workflow process. We also obtained and reviewed Accela's 
Service Organization Control (SOC) 2 report, conducted in accordance with the Statement on Standards for 
Attestation Engagement No. 16 (SSAE-16) issued by the American Institute of Certified Public Accountants 
(AICPA). SOC 2 reports provide an independent attestation relating to the security, availability, processing 
integrity, confidentiality, and privacy of an information system, such as the Accela platform. Finally, we 
obtained information from the public, including public comments provided at City of Los Angeles Cannabis 
Regulation Commission meetings, and information provided directly to the audit team. 
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In order to determine if any Applicant accessed the Accela platform or the Phase 3 application prior to 
September 3, 2019, at 10:00 am (i.e., the “Application Time”) and, if so, the factors that enabled the 
applicant(s) to gain access prior to the Application Time, we obtained, reviewed, and analyzed audit logs 
from the Accela system. These logs identified key timestamps related to Applicant sign-on attempts, 
application start times, application submit times, document upload times, and various timestamps 
associated with the Phase 3 Retail Round 1 application workflow. We analyzed these timestamps with the 
purpose of evaluating internal consistency, identifying any indicators of non-human intervention in the 
application process, and mapping a timeline identifying individual account activation, sign-on to the Accela 
portal, start of each application, document upload times, and application submittal. 


In order to evaluate the measures taken by the Department to address any instances in which Applicants 
accessed the Accela platform or the Phase 3 Application prior to the Application Time, and to determine the 
extent to which Applicants were impacted by any instances in which Applicants accessed Accela prior to 
the Application Time, we obtained and analyzed documentation of the Department's normalization process 
employed to determine and identify the ‘Top 100” Applicants for processing. This included an analysis of 
the accuracy of the calculations performed, as well as the Department's choice of timestamps used for the 
normalization process. 


Audit work was performed from January 2020 through March 2020. A draft of this report was provided to 
Department management for review and discussion and an Exit Conference was held on March 25, 2020, 
The Department agreed with the findings and recommendations presented in this report, and the responses 
and feedback provided by management were considered and incorporated where appropriate in the final 
report. We appreciate the cooperation provided by the City, Department, and Accela during the course of 
this audit. 


We conducted this performance audit in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate 
evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We 
believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on 
our audit objectives. 
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Audit Findings & Recommendations 


As described previously, the Department provided informational materials to the public detailing how 
potential Applicants were to sign onto the Accela Civic Platform, navigate to the application page, and 
complete and submit the application for a Phase 3 Retail Round 1 processing. Spurred by concerns raised 
by the public regarding the potential that certain Applicants accessed the system early and thereby gained 
an advantage over other Applicants, the CAO's Office asked our firm to determine if anyone accessed the 
system prior to the Application Time; if so, who gained early access and what factors enabled them to gain 
early access. And, if individuals did gain early access, what did the Department do to mitigate the impact 
such access would have had on the pool of Applicants, and based on these efforts, is there reason to 
believe that any Applicant benefited from accessing the system early. The remainder of our report 
represents our conclusions to the CAO's objectives. 


Objective #1 


Establish a comprehensive understanding as to whether any Applicants accessed Accela prior to 
the Application Time and, if so, (1) how many Applicants, (2) which Applicants, and (3) the times at 
which they accessed Accela. 


The Department designated 10:00 AM PDT on September 3, 2019, as the “Application Time.” We examined 
system audit logs that identified every attempted login to the Department's “LADCR" module of the Accela 
Civic Platform between August 28, 2019, when Accela disabled user accounts, and September 3, 2019, 
when Accela re-enabled user accounts and opened the Application Window. We also examined system 
audit logs that identified every instance in which a user started an application during this same period. We 
found the following: 


e 226 Applicants signed on to the Accela Civic Platform before 10:00 Am on September 3, 2019. A 
list of the Applicants that accessed Accela Civic Platform prior to 10:00 Amis provided in Appendix 
A of this report. 


e 224 of these Applicants signed on between 9:59:46 Amand 9:59:59 Am, and two Applicants signed- 
on at 9:48:44 AM and 9:49:54 AM. 


e Ofthe Applicants that signed onto the Accela Civic Platform prior to 10:00 AM on September 3, 
2019, 14 accessed the Phase 3 Retail Round 1 application prior to 10:00 Am. These Applicants are 
presented in Exhibit 3. 
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EXHIBIT 3: APPLICATIONS STARTED PRIOR TO THE APPLICATION WINDOW 















































Record Number Application Start Time 
LA-C-19-310001-R-APP 9:50:12 
LA-C-19-310003-R-APP 9:56:03 
LA-C-19-310004-R-APP 9:59:54 | 
LA-C-19-310090-R-APP 9:59:54 
LA-C-19-310006-R-APP 9:59:56 
LA-C-19-310009-R-APP 9:59:56 
LA-C-19-310126-R-APP 9:59:56 
LA-C-19-310022-R-APP 9:59:57 
LA-C-19-310087-R-APP 9:59:57 
LA-C-19-310013-R-APP 9:59:58 | 
LA-C-19-310025-R-APP 9:59:59 
LA-C-19-310027-R-APP 9:59:59 
LA-C-19-310059-R-APP 9:59:59 
LA-C-19-310125-R-APP 9:59:59 








Objective #2 


Determine the root cause of any instances in which Applicants accessed Accela prior to the 
Application Time. 


In order to determine the root cause of the identified instances in which Applicants accessed the Accela 
Civic Platform, we reviewed application performance monitoring data and application logs, and the 
processes employed to restrict and then re-enable system access by disabling and enabling user accounts. 
We found that Accela provided application performance monitoring data and application logs to corroborate 
their assertions that no bot-like behavior was observed on September 3, 2019. Although Accela could not 
provide all the supporting data and system logs that were the basis of their forensic analysis, the data 
provided generally supports Accela's assertions because the activity during September 3, 2019 does not 
appear to indicate any nefarious activities occurred. 


The root cause of the early access identified in Objective #1, however, was the result of the technique 
employed to restrict user access to the Accela Civic Platform by disabling user accounts. The process of 
re-enabling user accounts granted all Applicants access to the Accela system starting at approximately 
9:59:46 Am, According to the Department and Accela, they re-enabled these accounts prior to 10:00 AM by 
design to ensure all Applicants could access the start of the application by 10:00 Am. Yet, there was no 
other system barrier preventing Applicants, once signed on, from navigating to the application page and 
Starting the application. As described earlier, Applicants needed to sign onto the Accela Civic Platform and 
navigate through several pages in order to start the application itself. It took Applicants about 32 seconds to 
navigate from the sign-on page to the actual application, as measured by the median time lapsed, though 
some navigated through each page in just a few seconds. Nearly all Applicants that signed onto the Accela 
Civic Platform prior to 10:00 Am did so because the Department and Accela allowed all Applicants to do so. 
In fact, 224 of the 226 Applicants that accessed the Accela system before 10:00 Am gained access through 
the planned sign-on process between 9:59:46 am and 9:59:59 AM. 
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While re-enabling user accounts granted access to the Accela Civic Platform by design, this technique was 
problematic in two ways. 


v First, user accounts can be disabled and enabled manually by authorized Accela and Department 
personnel. As described previously, the Department required Applicants who were unable to 
access their accounts to come to the Department's office in person to request a manual reset to 
the Applicant's user account and password. The ability of management to enable and disable 
accounts in Accela's Civic Platform, and the Department's requirement for Applicant's to request 
account resets in person, introduced the potential for human error. 


The morning of September 3, 2019, several Applicants arrived at the Department's office to 
request a manual reset of their accounts and passwords. The Department, consistent with its 
mission to remove barriers to the application process for all Social Equity Applicants, fulfilled the 
request to reset user accounts for as many Applicants as possible. Doing so, however, required the 
Department to enable each Applicant's account so that the Applicant could sign-on with the 
temporary password they were given and to create a permanent password that the Applicant could 
then use to sign on and start the application. As illustrated in Exhibit 4, nine (9) Applicant accounts 
were enabled prior to 10:00 AM on September 3, 2019, as aresult of this process. The resets 
occurred between 07:42:40 and 09:35:00. According to Department management, it did not occur 
to them at the time that they should have re-disabled the Applicant's accounts after the passwords 
were reset; it was only with hindsight that the Department realized that its actions allowed a few 
Applicants to access the Accela Civic Platform before all other Applicant accounts were re- 
enabled. 


EXHIBIT 4; APPLICANTS WHOSE ACCOUNTS WERE MANUALLY ENABLED 


























Applicant Re-Enabled 
PUBLICUSER294886 9/3/2019 7:42:40 
PUBLICUSER293622 | 9/3/2019 7:46:58 
PUBLICUSER293999 | 9/3/2019 7:48:46 
PUBLICUSER299988 | 9/3/2019 9:00:39 
PUBLICUSER304026 | 9/3/2019 9:02:17 
PUBLIGUSER240822 | 9/3/2019 9:04:40 | 
PUBLICUSER296105 | 9/3/2019 9:06:59 
PUBLICUSER279855 | 9/3/2019 9:08:13 | 
PUBLICUSER304054 | 9/3/2019 9:35:00 | 








Seven (7) of these nine Applicants did not access the application early and thus gained no benefit 
from the Department re-enabling their accounts. Two (2) Applicants, however, did sign on prior to 
Accela's script to re-enable user accounts (see the shaded rows in Exhibit 4). These two 
Applicants are identified as the first two sign on the Accela Portal in Appendix A, and both began 
their applications prior to 9:59 AM, as illustrated in Exhibit 7. The potential benefit these two (2) 
Applicants would have gained by accessing the application early was negated by the Department's 
“normalization” process, as described in Objective #3 below. 
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Y Second, the technique of disabling accounts and then re-enabling them immediately before the 
scheduled Application Window obfuscated what should have been very clear. The Phase 3 Retail 
Round 1 application process was a race, and the execution of this technique, re-enabling user 
accounts within 14 seconds of Application Time, created the potential for confusion regarding the 
starting line. 


In the month leading up to the September 3 application start time, the Department routinely 
referred to the “Application Window” as commencing at 10:00 AM on September 3, 2019. For 
instance, on August 19, 2019, the Department issued a statement indicating that it was “pleased to 
announce the opening of Phase 3 Round 1 Application Window beginning on Sept. 3, 2019 at 
10:00 Am.” This appears clear: Applicants would be able to access and begin working on the 
application at this time. 


However, the Department issued other statements that, in hindsight, differ from this message and 
could reasonably have influenced potential Applicants’ understanding of the application start time. 
Specifically, on August 26, 2019, the Department released this message: “The Accela portal will be 
unavailable from Wednesday, August 28, 10 AM PDT until Tuesday, September 3, 2019, 10 AM 
PDT while it undergoes maintenance in preparation for the opening of the application window." The 
Accela portal was indeed open before 10:00 Am, specifically by 9:59:46 Am, and Applicants were 
able to navigate to the application page during this time. 


The Department's decision to disable all user accounts and then to re-enable them at the same 
time the Department intended to open the “Application Window’ could reasonably lead Applicants 
to believe that they could not sign on to the system until 10:00 Am. Apart from the two early 
Applicants identified in this report, all other Applicants that attempted to sign into the system prior 
to 9:59:46 am were denied access and met with an alert that “An error has occurred. The Accela 
System is currently down for scheduled maintenance. Phase 3 Round 1 is scheduled to go live on 
September 3%, 2019 at 10 AM Pacific Daylight Time (PDT).” While 224 Applicants did access the 
system during the 14 seconds prior to 10:00 Am, system sign-on logs suggest more Applicants 
waited until 10:00 AM. As shown in Exhibit 5, audit log data shows that 321 Applicants signed-on to 
the Accela portal between 10:00:00 Am and 10:00:59 AM, 154 of whom did so within the first two 
seconds of the hour. Ultimately, evidence suggests that some Applicants attempted to sign-on as 
early as possible while other Applicants waited until exactly 10:00 Am to sign-on because of what, 
in hindsight, was imprecise messaging by the Department. 
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EXHIBIT 5. APPLICANT SIGN-ON FREQUENCY AT APPLICATION TIME (9:59-10:00 AM) 


100 


30 


20 
4 


1456:47 :48:49 -50 151 :52 :53 :54 :55 :56 :57 :58 59:00:01 :02 :03 :04::05 :06 :07 :08 :09 :10:11:12 113 114 


39 :00 


Based on this, we found the method used to roll out the application to be confusing and prone to human 
error. Instead, the Department and Accela could have used an alternative method that technically restricted 
all users from accessing the application without deactivating accounts or resetting passwords, and made 
the Accela Civic Platform available to all users well before the Application Window so as not to conflate the 
Application Window with the platform sign-on process. This would be analogous to Southwest Airlines 
enabling individuals to log into their accounts but restricting their ability to check in for their flight until 
exactly 24 hours prior to their flight time. This would have prevented the type of human error that led to the 
advanced enabling of user accounts, and it would have better ensured all Applicants could access the 
application simultaneously. According to Accela, it elected the method it did to ensure the Department and 
Accela staff had access to the production environment during the shut-down period of August 28 through 
September 2, 2019, but acknowledges in hindsight that an alternative approach would have resulted in a 
better user experience. 


Therefore, in order to avoid similar problems in the future, we recommend that, when establishing 
upcoming Application Windows, the Department of Cannabis Regulation develop a method that restricts all 
users from accessing the application without deactivating accounts or resetting passwords, and make the 
Accela Civic Platform available to all users well before the Application Window so as not to conflate the 
Application Window with the platform sign-on process. 


Objective #3 


Assess any measures taken by the Department to address any instances in which Applicants 
accessed Accela prior to the Application Time. 


To negate any advantage gained by the 14 Applicants who accessed the Phase 3 Retail Round 1 
application prior to the 10:00 Am, the Department performed a timestamp “normalization” process. This 
consisted of calculating the length of time taken by an Applicant to complete their application and adding 
that time to the planned application start time of 10:00 aM to arrive at a normalized application submit time. 
As shown in Exhibit 6, the Department used the time at which an Applicant entered into the Application 
from the Terms and Conditions page as the start time for the normalization calculation, and the application 
submit time as the end time. As a result, the normalized time represented the time at which the application 
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would have been submitted had the Applicant started the application at 10:00 AM, thereby neutralizing the 
advantage of an early application start time. 


EXHIBIT 6. THE PHASE 3 RETAIL ROUND 1 NORMALIZATION PROCESS 


Application 
Document Submitted 







Applicat Upload 
pplication 
Terms & 
Conditions 

Apply 
Dashboard 


Time Start of Application 
Point of Normalization 


To evaluate the Department's normalization process, we interviewed key Department and Accela staff 
involved in the process, and independently reviewed and validated the Department's normalization 
calculations using application start and end times taken from Accela's system audit logs. In addition, we 
assessed whether the start and end times used by the Department to perform the normalization process 
were reasonable based on factors including the process workflow, system configuration, and how the 
Department communicated the process to prospective Applicants leading up to the Application Window. 


As shown in Exhibit 7, for each of the 14 Applicants accessing the application prior to 10:00 Am, 
normalization had the effect of shifting their application submit time later. Because the Department 
reviewed applications in order of time submitted, the normalization process changed the order in which 
applications would be reviewed. The submit times for applications that were started at or after 10:00 AM 
were not changed by the normalization process. Our review found the timestamps used by the Department 
for normalization matched those in Accela's system logs and the Department's calculation of application 
duration and normalization to be consistent and accurate. 
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EXHIBIT 7: RESULTS OF THE DEPARTMENT'S NORMALIZATION PROCESS 


Actual 
Submit Time 




















Actual 
Submit Order* 


Application 
Record # Start Time 




































































LA-C-19-310001-R-APP 55 
LA-C-19-310003-R-APP 9:56:03 179 
LA-C-19-310004-R-APP 9:59:54 | 1 
LA-C-19-310090-R-APP 9:59:54 101 
LA-C-19-310006-R-APP 9:59:56 2 
LA-C-19-310009-R-APP 9:59:56 10:01:01 7 10:01:05 10 
LA-C-19-310126-R-APP_| 9:59:56 10:01:45 118 10:01:49 139 
LA-C-19-310022-R-APP 9:59:57 10:01:11 20 10:01:14 21 
LA-C-19-310087-R-APP 9:59:57 10:01:34 82 10:01:37 88 
LA-C-19-310013-R-APP 9:59:58 10:01:03 9 10:01:05 11 
LA-C-19-310025-R-APP 9:59:59 10:01:13 23 10:01:14 21 
LA-C-19-310027-R-APP 9:59:59 10:01:13 23 10:01:14 21 
LA-C-19-310059-R-APP 9:59:59 10:00:40 56 10:01:28 56 
LA-C-19-310125-R-APP 9:59:59 10:01:45 118 10:01:46 123 





* Represents the lowest possible order for the submit time, given that multiple applications can share the same time. 


While this normalization process focused on any instances in which Applicants accessed the application 
prior to the Application Window, the audit objective set forth by the CAO requires an evaluation of the 
measures taken by the Department to address any instances in which Applicants accessed Accela prior to 
the Application Window. This can only be measured by the sign-on timestamp, and the Department did not 
normalize application submittal results based on an Applicant's access to the Accela Civic Platform 
precisely because Accela “opened the doors” to the platform prior to 10:00 Am by design, as discussed 
previously. This was consistent with the Department's intended opening of the Application Window time 
and was based on an application start time defined as the time at which an Applicant clicked “Continue” on 
the Terms and Condition page to enter into the application. 


Objective #4 


Assess how any Applicants were impacted, if at all, by any instances in which Applicants accessed 
Accela prior to the Application Time and any measures taken by the Department to address such 
instances. 


Based on the analysis described in Objective #3, we found the application timestamp normalization process 
performed by the Department effectively negated any benefits for all 14 Applicants that accessed the 
application before 10:00 Am. We further found the Department's approach to normalization to be reasonable 
and appropriate, and reflected the intent of the program to process applications on a first come, first served 
basis as measured by application timestamps. Finally, we found that there is no evidence to suggest that 
Department's process of activating user accounts benefited any users over any other users, with the 
exception of the two Applicants whose passwords were manually reset and who started the application 
prior to Accela's re-enabling of all other accounts—the benefit of which was again negated by the 
Department's normalization process. All accounts were activated using a single script shortly prior to 10:00 
Am. However, the potential misperception among some Applicants that they were prohibited from signing on 
to the Accela Civic Platform until 10:00 AM may have placed those Applicants at a disadvantage. Those 
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Applicants that waited until 10:00 am to sign on would have lost time in the process; as mentioned above, 
system audit logs show that, among all Applicants, the median length of time it took an Applicant to 
navigate from the platform sign-on to the application page was 32 seconds. It is only in hindsight and with 
this data and related analysis that an evaluation of an alternative normalization process utilizing sign-on 
times instead of application start times becomes relevant. 


To determine the magnitude of the impact that a normalization of sign-on timestamps could have had on 
the outcome of the application process, we re-evaluated the rank order based on normalized Applicant sign 
on times and found that nearly 90 percent of the Applicants that finished in the ‘top 100” remained in the 
“top 100.” However, our analysis of system audit log data shows that as many as 11 Applicants whose 
applications were determined to be ineligible may have qualified for further processing if the Department 
normalized application submission times based on sign-on times. This could affect 11 applicants, whose 
applications have already been approved. In all but one of instance, the impact results solely from the 
change moving some applications in and other applications out of the first 100 qualified applications. In one 
instance, the change to the order reviewed led to an application leaping ahead of the very application that 
had caused it to be deemed ineligible due to program rules prohibiting licensed businesses from operating 
within 700 feet of one another. The 11 Applicants deemed eligible for further processing that may be 
impacted are shown in Exhibit 8. 


EXHIBIT 8: PREVIOUSLY INVOICED APPLICATIONS POTENTIALLY IMPACTED BY SIGN-ON TIME NORMALIZATION PROCESS 




















Normalized 
Submit Time 


10:01:28 


Sign-On Normalized 
Submit Time 


Actual 
Ka Sign-On Time 
LA-C-19-310001-R-APP 
LA-C-19-310118-R-APP 
LA-C-19-310111-R-APP 
LA-C-19-310125-R-APP 
LA-C-19-310114-R-APP 
LA-C-19-310113-R-APP 
LA-C-19-310122-R-APP 





























10:01:42 
10:01:45 10:01:54 


10:01:52 





























| 

| 
LA-C-19-310116-R-APP = | 10:01:43 10:01:52 
LA-C-19-310133-R-APP | 10:01:46 10:01:54 
LA-C-19-310134-R-APP | 10:01:46 10:01:52 
LA-C-19-310047-R-APP | 10:01:23 10:01:29 








Though our review found that as many as 11 other applications could have been reviewed and may have 
been deemed eligible for further processing following normalization based on sign-on times, no definitive 
conclusions can be drawn at this time. While all accepted applications were subject to the same review 
process to determine eligibility and cross-application impacts, the Department has not completed such a 
review of applications that were not within the top 100. Therefore, this audit cannot conclude definitively on 
those alternative applications that may have been deemed eligible for further processing, nor does this 
audit make any recommendation regarding the outcome of the Phase 3 Retail Round 1 process. 


Rather, we found that while imprecise messaging and an ill-suited technique to eliminate access to the 
Phase 3 Retail Round 1 application prior to the Application Window led to some confusion regarding the 
application process, there is no evidence that the Department's process to re-enable user accounts 
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benefited any users over any other users, and the Department's normalization process effectively negated 
any benefits for all 14 Applicants that accessed the application before 10:00 am. Our review found the 
Department's normalization of the application start time to be reasonable and appropriate, and reflected the 
intent of the program to process applications on a first come, first served basis as measured by application 
timestamps. Despite our conclusion that the resulting confusion lends legitimacy to an alternative 
normalization process, we found the Department's approach to normalization to be reasonable and 
appropriate, and we did not find alternative methods to be more reasonable or more appropriate than the 
adopted approach, 
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Appendix A — Applicant Sign-On Timestamps Prior to 10:00 am 

























































































User Initial Sign-On Application ID 
4 | PUBLICUSER240822 | 9/3/2019 9:48:44 AM LA-C-19-310001-RAPP 
2 | PUBLIGUSER293999 | 9/3/2019 9:49:54 AM LA-C-19-310003-R-APP 
3 | PUBLICUSER291321 | 9/9/2019 9:59:48 AM LA-0-19:310004-R-APP 
4 | PUBLIGUSER291987 | 913/2019 9:59:46 AM 1A:0-19:310274-R-APP 
5 | PUBLIGUSER296723 | 9/3/2019 9:59:46 AM LAC-19-310294-R-APP 
6 | PUBLICUSER299071 | 913/2019 9:59:46 AM LAC-19-310017-R-APP 
7 | PUBLIGUSER299350 | 9/3/2019 9:59:46 AM LAC-19-310253-R-APP 
8 | PUBLIGUSER301325 | 9/3/2019 9:59:46 AM LA-C-19-310427-R-APP 
9 | PUBLIGUSER301469 | 9/3/2019 9:59.46 AM LAC-19-310231-R-APP 
10 | PUBLICUSER302055 | 9/3/20199:59:46 AM | LA-C-19-310739-R-APP, LA-C-19-310750-R-APP 
11 | PUBLICUSER302548 | 9/9/2019 9:59:46 AM LA-C-19-310009-R-APP 
12 | PUBLIGUSER303614 | 9/9/2019 9:59:46 AM LA-C-19-310090-R-APP 
13 | PUBLIGUSER304056 | 9/9/2019 9:59:46 AM LAC-19-310130-R-APP 
14 | PUBLICUSER241281 | 9/3/2019 9:59:47 AM LA-C-19-310867-R-APP 
15 | PUBLIGUSER246021 | 9/9/2019 9:59:47 AM LAC-19-310022-R-APP 
16 | PUBLIGUSER285041 | 9/3/2019 9:59:47 AM LA-C-19-310404-R-APP 
17 | PUBLIGUSER289477 | 913/2019 9:59:47 AM LAC-19-310175-R-APP 
18 | PUBLIGUSER289557 | 913/2019 9:59:47 AM LA-P-19:310291-R-APP 
19 | PUBLIGUSER289621 | 913/2019 9:59:47 AM LA-0-19:310040-R-APP 
20 | PUBLICUSER289826 | 3201995947 AM | Aa yen nde 
21 | PUBLICUSER289680 | 9/3/2019 9:59:47 AM LA.C-19-310014-R-APP 
22 | PUBLICUSER290039 | 913120199:59:47 AM | LA-C-19-310045-R-APP, LA-C-19-310310-R-APP 
23 | PUBLICUSER291166 | 9/3/2019 9:50:47 AM LAG-19-310263-R-APP 
24 | PUBLICUSER295289 | 9/3/2019 9:59:47 AM ENT A TRASSI ye 
25 | PUBLICUSER296066 | 9/3/2019 9:59:47 AM LA.C-19-310370-R-APP 
26 | PUBLIGUSER297774 | 9/3/2019 9:59:47 AM LA-C-19-310007-R-APP 
27 | PUBLICUSER298061 | 9/3/2019 9:59:47 AM LA-C-19-310369-R-APP 
28 | PUBLICUSER298480 | 9/3/2019 9:59:47 AM TA.0-19-310224-R-APP 
29 | PUBLICUSER299632 | 9/3/2019 9:59:47 AM TAC-19-310163-R-APP 
30 | PUBLICUSER302332 | 9/3/2019 9:59:47 AM LA-G-19-310093-R-APP 
31_| PUBLICUSER302966 | 9/3/2019 9:59:47 AM LAC-19-310285-R-APP 
32 | pugiiousekasssas | roto asos A | LAC-19-3T0351-R-APP, LA.C-19-310529R-APP, 














LA-C-19-310572-R-APP, LA-C-19-310686-R-APP 
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User Initial Sign-On Application ID 




















33 | PUBLICUSER288641 9/3/2019 9:59:48 AM LA-C-19-310052-R-APP, LA-C-19-310413-R-APP 
34 | PUBLICUSER288946 9/3/2019 9:59:48 AM LA-C-19-310091-R-APP 
35 | PUBLICUSER289179 9/3/2019 9:59:48 AM LA-C-19-310255-R-APP 
36 | PUBLICUSER289426 9/3/2019 9:59:48 AM LA-C-19-310006-R-APP 
37 | PUBLICUSER289677 9/3/2019 9:59:48 AM LA-C-19-310144-R-APP 
38 | PUBLICUSER289862 9/3/2019 9:59:48 AM LA-C-19-310089-R-APP 
39 | PUBLICUSER289951 9/3/2019 9:59:48 AM LA-C-19-310302-R-APP 





LA-C-19-310030-R-APP, LA-C-19-310059-R-APP, 


40 | PUBLICUSER290311 9/3/2019 9:59:48 AM LA-C-19-310361-R-APP 

























































































41 | PUBLICUSER290868 | 9/3/2019 9:59:48 AM LAP-19-310399-R-APP 

42 | PUBLICUSER201696 | 9/3/2019 9:59:48 AM TAC-19-310331-R-APP 

43 | PUBLICUSER294497 | 9/3/2019 9:59:48 AM 1A:0-19:310041-R-APP 

44 | PUBLICUSER294503 | 9/3/2019 9:59:48 AM LA-0-19-310086-R-APP 

45 | PUBLICUSER208073 | 9/3/2019 9:59:48 AM LAC-19-310266-R-APP 

46 | PUBLICUSER303342 | 9/9/2019 9:59:48 AM LAC-19-310177-R-APP 

47 | PUBLICUSER212021 | 9/3/2019 9:59:49 AM LA-C-19-310087-R-APP 

48 | PUBLICUSER237928 | 9/3/2019 9:59:49 AM LAG-19-310166-R-APP 

49 | PUBLIGUSER289394 | 9/3/2019 9:59:49 AM LA-C-19-310111-R-APP 

50 | PUBLICUSER290870 | 9/3/2019 9:59:49 AM LAC-19-310433-R-APP 

51 | PUBLICUSER291285 | 9/3/2019 9:59:49 AM LAC-19-310010-R-APP 

52 | PUBLIGUSER294884 | 9/3/2019 9:59:49 AM LAC-19-310051-R-APP 
PUBLICUSER296052 59. z P, LA-C-19-310234-R-APP 

54 | PUBLICUSER296101 | 9/3/2019 9:59:49 AM LA-C-19-310544-R-APP 

55 | PUBLICUSER297679 | sa2o199:59:49AM | LAC ie ee RERI 

56 | PUBLICUSER297954 | 9/3/2019 9:59:49 AM LA-C-19-310376-R-APP 

57 | PUBLICUSER298845 | 9/3/2019 9:59:49 AM LAC-19-310170-R-APP 

58 | PUBLICUSER299461 | 9/3/2019 9:59:49 AM LA-0-19-310078-R-APP 

59 | PUBLIGUSER299986 | 9/3/2019 9:59.49 AM LAC-19-310013. RAPP 

60 | PUBLICUSER302305 | 9/3/2019 9:59.49 AM LAC-19-310118-R-APP 

61 | PUBLICUSER303325 | 9/3/2019 9:59:49 AM LA-C-19-310046-R-APP 

62 | PUBLICUSER303385 | 9/3/2019 9:59:49 AM LA-C-19:310042-R-APP 

63 | PUBLICUSER304096 | 9/3/2019 9:59:49 AM LA-C-19-310406-R-APP 

64 | PUBLICUSER237990 | 9/3/2019 9:59:50 AM LAC-19-310114-R-APP 

65 | PUBLIGUSER238845 | 9/3/2019 9:59:50 AM LAC-19-310199-R-APP 

66 | PUBLICUSER280250 | 9/3/2019 9:59:50 AM LAC-19-310032.R- APP 

67 | PUBLICUSER289299 | 9/3/2019 9:59:50 AM LA0-19:310127-R-APP 
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User Initial Sign-On Application ID 




















































































































68 | PUBLICUSER289671 9/3/2019 9:59:50 AM LA-C-19-310057-R-APP 
69 | PUBLICUSER289993 9/3/2019 9:59:50 AM LA-C-19-310020-R-APP 
70 | PUBLICUSER290913 9/3/2019 9:59:50 AM LA-C-19-310031-R-APP 
71 PUBLICUSER291040 9/3/2019 9:59:50 AM LA-C-19-310374-R-APP 
72 | PUBLICUSER292338 9/3/2019 9:59:50 AM LA-C-19-310035-R-APP 
73. | PUBLICUSER293028 9/3/2019 9:59:50 AM LA-C-19-310277-R-APP 
74 | PUBLICUSER293225 9/3/2019 9:59:50 AM LA-C-19-310289-R-APP 
75 | PUBLICUSER293621 9/3/2019 9:59:50 AM LA-C-19-310460-R-APP 
76 | PUBLICUSER293623 9/3/2019 9:59:50 AM LA-C-19-310216-R-APP 
77. | PUBLICUSER293627 9/3/2019 9:59:50 AM LA-C-19-310160-R-APP 
78 | PUBLICUSER294029 9/3/2019 9:59:50 AM LA-P-19-310325-R-APP 
79 | PUBLICUSER294052 9/3/2019 9:59:50 AM LA-C-19-310113-R-APP 
80 | PUBLICUSER294117 9/3/2019 9:59:50 AM LA-C-19-310194-R-APP 
81 PUBLICUSER294163 9/3/2019 9:59:50 AM LA-C-19-310319-R-APP 
82 | PUBLICUSER294164 9/3/2019 9:59:50 AM LA-C-19-310248-R-APP 
83 | PUBLICUSER294358 9/3/2019 9:59:50 AM LA-C-19-310193-R-APP 
84 | PUBLICUSER294454 9/3/2019 9:59:50 AM LA-C-19-310049-R-APP 
85 | PUBLICUSER294456 9/3/2019 9:59:50 AM LA-C-19-310029-R-APP 
86 | PUBLICUSER294644 9/3/2019 9:59:50 AM LA-C-19-310088-R-APP 
87 | PUBLICUSER294703 9/3/2019 9:59:50 AM LA-C-19-310044-R-APP 
88 | PUBLICUSER294747 9/3/2019 9:59:50 AM LA-C-19-310365-R-APP 
89 | PUBLICUSER294755 9/3/2019 9:59:50 AM LA-C-19-310060-R-APP 
90 | PUBLICUSER294863 9/3/2019 9:59:50 AM LA-C-19-310039-R-APP, 
91 PUBLICUSER295025 9/3/2019 9:59:50 AM LA-C-19-310181-R-APP 
92 | PUBLICUSER295445 9/3/2019 9:59:50 AM LA-C-19-310192-R-APP 
93 | PUBLICUSER295642 9/3/2019 9:59:50 AM LA-C-19-310079-R-APP 
94 | PUBLICUSER295729 9/3/2019 9:59:50 AM LA-C-19-310158-R-APP 
95 | PUBLICUSER295988 9/3/2019 9:59:50 AM LA-C-19-310025-R-APP 
96 | PUBLICUSER296269 9/3/2019 9:59:50 AM LA-P-19-310480-R-APP 
97 | PUBLICUSER296298 9/3/2019 9:59:50 AM LA-C-19-310048-R-APP 
98 | PUBLICUSER298975 9/3/2019 9:59:50 AM LA-C-19-310141-R-APP 
99 | PUBLICUSER299244 9/3/2019 9:59:50 AM LA-C-19-310227-R-APP 
100 | PUBLICUSER301281 9/3/2019 9:59:50 AM LA-C-19-310256-R-APP 
101 | PUBLICUSER302947 9/3/2019 9:59:50 AM LA-C-19-310147-R-APP, 
102 | PUBLICUSER303981 9/3/2019 9:59:50 AM LA-C-19-310125-R-APP 
103 | PUBLICUSER304100 9/3/2019 9:59:50 AM LA-C-19-310073-R-APP 
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User Initial Sign-On Application ID 
104 | PUBLICUSER304167 9/3/2019 9:59:50 AM LA-C-19-310126-R-APP 
105 | PUBLICUSER304254 9/3/2019 9:59:50 AM LA-C-19-310026-R-APP, 





106 | PUBLICUSER205241 


9/3/2019 9:59:51 AM 


LA-C-19-310536-R-APP, LA-P-19-310722-R-APP 





107 | PUBLICUSER236994 


9/3/2019 9:59:51 AM 


LA-C-19-310012-R-APP 





108 | PUBLICUSER289757 


9/3/2019 9:59:51 AM 


LA-C-19-310112-R-APP 





109 | PUBLICUSER291398 


9/3/2019 9:59:51 AM 


LA-C-19-310136-R-APP 





110 | PUBLICUSER291581 


9/3/2019 9:59:51 AM 


LA-C-19-310204-R-APP 





111. | PUBLICUSER292306 


9/3/2019 9:59:51 AM 


LA-C-19-310153-R-APP 





112 | PUBLICUSER292696 


9/3/2019 9:59:51 AM 


LA-C-19-310451-R-APP 





113. | PUBLICUSER292755 


9/3/2019 9:59:51 AM 


LA-C-19-310164-R-APP 





114 | PUBLICUSER293026 


9/3/2019 9:59:51 AM 


LA-C-19-310142-R-APP 














115 | PUBLICUSER293902 | 9/3/2019 9:59:51 AM LA-C-19-310278-R-APP 
116 | PUBLICUSER294112 | 9/3/2019 9:59:51 AM LA-C-19-310400-R-APP 
117 | PUBLICUSER294165 | 9/3/2019 9:59:51 AM LA-C-19-310116-R-APP 
118 | PUBLICUSER294213 | 9/3/2019 9:59:51 AM LA-C-19-310062-R-APP 





119 | PUBLICUSER294452 


9/3/2019 9:59:51 AM 


LA-C-19-310140-R-APP 





120 | PUBLICUSER294750 
121 | PUBLICUSER295446 


9/3/2019 9:59:51 AM 
9/3/2019 9:59:51 AM 


LA-C-19-310377-R-APP 
LA-C-19-310008-R-APP 





122 | PUBLICUSER295519 


9/3/2019 9:59:51 AM 


LA-C-19-310279-R-APP 





123 | PUBLICUSER295568 
124 | PUBLICUSER297389 





9/3/2019 9:59:51 AM 
9/3/2019 9:59:51 AM 


LA-C-19-310178-R-APP 
LA-C-19-310689-R-APP 





125 | PUBLICUSER297546 


9/3/2019 9:59:51 AM 


LA-C-19-310402-R-APP 





126 | PUBLICUSER297772 


9/3/2019 9:59:51 AM 





127 | PUBLICUSER299988 


9/3/2019 9:59:51 AM LA-C-19-310018-R-APP 


LA-C-19-310414-R-APP 





128 | PUBLICUSER300442 


9/3/2019 9:59:51 AM 


LA-C-19-310244-R-APP 





129 | PUBLICUSER301140 


9/3/2019 9:59:51 AM 


LA-C-19-310293-R-APP 





130 | PUBLICUSER302623 


9/3/2019 9:59:51 AM 


LA-C-19-310233-R-APP, LA-C-19-310668-R-APP 





131 | PUBLICUSER302703 


9/3/2019 9:59:51 AM 


LA-C-19-310191-R-APP 





132 | PUBLICUSER303427 


9/3/2019 9:59:51 AM 


LA-C-19-310334-R-APP 





133 | PUBLICUSER303465 


9/3/2019 9:59:51 AM 


LA-C-19-310053-R-APP, LA-C-19-310345-R-APP, 
LA-C-19-310575-R-APP 





134 | PUBLICUSER303555 


9/3/2019 9:59:51 AM 


LA-C-19-310219-R-APP 









































135 | PUBLICUSER303703 9/3/2019 9:59:51 AM LA-C-19-310122-R-APP 
136 | PUBLICUSER304021 9/3/2019 9:59:51 AM LA-C-19-310205-R-APP, LA-C-19-310660-R-APP 
137 | PUBLICUSER304237 9/3/2019 9:59:51 AM LA-C-19-310102-R-APP 
138 | PUBLICUSER289418 9/3/2019 9:59:52 AM LA-C-19-310354-R-APP 
139 | PUBLICUSER289494 9/3/2019 9:59:52 AM LA-C-19-310347-R-APP 
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User Initial Sign-On Application ID 
140 | PUBLICUSER289672 9/3/2019 9:59:52 AM LA-C-19-310415-R-APP 
141 | PUBLICUSER290045 9/3/2019 9:59:52 AM LA-C-19-310262-R-APP 
142 | PUBLICUSER290849 9/3/2019 9:59:52 AM LA-C-19-310105-R-APP 
143 | PUBLICUSER292154 9/3/2019 9:59:52 AM LA-C-19-310162-R-APP 
144 | PUBLICUSER292449 9/3/2019 9:59:52 AM LA-C-19-310350-R-APP 
145 | PUBLICUSER292664 9/3/2019 9:59:52 AM LA-C-19-310349-R-APP 
146 | PUBLICUSER292918 9/3/2019 9:59:52 AM LA-C-19-310342-R-APP 
147. | PUBLICUSER294099 9/3/2019 9:59:52 AM LA-C-19-310343-R-APP 
148 | PUBLICUSER295076 9/3/2019 9:59:52 AM LA-C-19-310471-R-APP 
149 | PUBLICUSER296395 9/3/2019 9:59:52 AM LA-C-19-310133-R-APP 
150 | PUBLICUSER296397 9/3/2019 9:59:52 AM LA-C-19-310036-R-APP 
151 | PUBLICUSER297052 9/3/2019 9:59:52 AM LA-C-19-310348-R-APP 
152 | PUBLICUSER299688 9/3/2019 9:59:52 AM LA-P-19-310804-R-APP 
153 | PUBLICUSER301222 9/3/2019 9:59:52 AM LA-C-19-310337-R-APP 
154 | PUBLICUSER301955 9/3/2019 9:59:52 AM LA-C-19-310318-R-APP 
155 | PUBLICUSER302862 9/3/2019 9:59:52 AM LA-C-19-310180-R-APP 
156 | PUBLICUSER303130 9/3/2019 9:59:52 AM LA-C-19-310341-R-APP 
157. | PUBLICUSER288150 9/3/2019 9:59:53 AM LA-C-19-310531-R-APP, LA-C-19-310740-R-APP 
158 | PUBLICUSER289230 9/3/2019 9:59:53 AM LA-C-19-310128-R-APP, LA-C-19-310870-R-APP 
159 | PUBLICUSER289448 9/3/2019 9:59:53 AM LA-C-19-310360-R-APP 
160 | PUBLICUSER289631 9/3/2019 9:59:53 AM LA-C-19-310150-R-APP 
161 | PUBLICUSER290031 9/3/2019 9:59:53 AM LA-C-19-310284-R-APP 
162 | PUBLICUSER291874 9/3/2019 9:59:53 AM LA-C-19-310081-R-APP 
163 | PUBLICUSER294702 9/3/2019 9:59:53 AM LA-C-19-310171-R-APP 
164 | PUBLICUSER296293 9/3/2019 9:59:53 AM LA-C-19-310200-R-APP 
165 | PUBLICUSER299446 9/3/2019 9:59:53 AM LA-C-19-310109-R-APP 
166 | PUBLICUSER301681 9/3/2019 9:59:53 AM LA-C-19-310011-R-APP 
167 | PUBLICUSER301746 9/3/2019 9:59:53 AM LA-C-19-310389-R-APP 
168 | PUBLICUSER303263 9/3/2019 9:59:53 AM LA-C-19-310330-R-APP 
169 | PUBLICUSER304110 9/3/2019 9:59:53 AM LA-C-19-310108-R-APP, LA-C-19-310697-R-APP 
170 | PUBLICUSER294883 9/3/2019 9:59:54 AM LA-C-19-310336-R-APP 
171 | PUBLICUSER237464 9/3/2019 9:59:54 AM LA-C-19-310210-R-APP 
172 | PUBLICUSER241209 9/3/2019 9:59:54 AM | LA-C-19-310134-R-APP, LA-C-19-310682-R-APP 
173 | PUBLICUSER242705 9/3/2019 9:59:54 AM LA-C-19-310211-R-APP, LA-C-19-310695-R-APP 
174. | PUBLICUSER289015 9/3/2019 9:59:54 AM LA-C-19-310338-R-APP 
175 | PUBLICUSER289415 9/3/2019 9:59:54 AM LA-C-19-310047-R-APP 
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User Initial Sign-On Application ID 
176 | PUBLICUSER290096 | 9/3/2019 9:59:54 AM LA-C-19-310326-R-APP 
177 | PUBLICUSER297309 | 9/9/2019 9:59:54 AM LAC-19-310431-RAPP 
178 | PUBLIGUSER301317 | 919/2019959:54AM | LA-6-19:310184-RAPP, LA-C-19-310687-R-APP 
479 | PUBLICUSER301682 | 9/3/2019 9:59:54 AM LA-C-19-310186-R-APP 
180 | PUBLICUSER301954 | 973/20199:59:54 AM | LA-C-19-310232-R-APP, LA-C-19-310706-R-APP 
181 | PUBLICUSER304101 | 9/3/2019 9:59:54AM_ | LAC-19-310237-R-APP, LA-C-19-310854-R-APP 
182 | PUBLICUSER304212 | 9/9/2019 9:59:54 AM LA-C-19-310096-R-APP 
483 | PUBLICUSER304264 | 973201995054 am | LAC-19 ahi SAKA AYE, 
184 | PUBLICUSER238475 | 9/3/2019 9:59:55 AM LA-0-19:310557-R-APP 
485 | PUBLICUSER280139 | 9/3/20199:59:55AM | LA-0-19:310196-RAPP, LA-0-19:310701-R-APP 
186 | PUBLICUSER290517 | 9/3/2019 9:59:55 AM 1A:0-19:310426-R-APP 
187 | PUBLICUSER291600 | 9/3/2019 9:59:55 AM 1A:0-19:310308-R-APP 
188 | PUBLIGUSER295940 | 913/20199:59:55AM | LAC-19-310428-R-APP, LA-0-19:310438-RAPP 
189 | PUBLIGUSER296384 | 9/3/2019 9:59:55 AM LAC-19-310251-R-APP 
190 | PUBLICUSER296503 | 9/3/2019 9:59:55 AM LAC-19-310028-R-APP 
CAC ABHOMZRAPP, LA. 19 310510 RAPP. 
191 | PUBLIGUBER238841 | 9/9/20199:58:55 AM | |. 0.19.310632.R-APP,LA-C-19-310699-R-APP, 
LA-C-19-310816-R-APP, LA-C-19-310817-R-APP 
192 | PUBLIGUSER300431 | 9/3120199:59:55 AM LA-0-19:310380-R-APP 
193 | PUBLICUSER302627 | 9/3/2019 9:59:55 AM LAP-19-310873-R-APP 
494 | PUBLICUSER303029 | 9/3/2019 9:59:55 AM LAC-19-310305-R-APP 
495 | PUBLICUSER304145 | 9/3/2019 9:59:55 AM LAC-19-310208-R-APP 
196 | PUBLICUSER289300 | 9/3/2019 9:59:46 AM LA-C-19-310542.R-APP 
197 | PUBLIGUSER290017 | 913/2019 9:59:56 AM LAC-19-310050-R-APP 
198 | PUBLICUSER292367 | 9/3/2019 9:59:56 AM LAC-19-310071-R-APP 
199 | PUBLIGUSER295364 | 9/3/2019 9:59:56 AM LA-0-19:310429-R-APP 
200 | PUBLICUSER289360 | 9/3/2019 9:59:57 AM LA-C-19-310320-R-APP, 
201 | PUBLICUSER296105 | 913/2019 9:59:57 AM LA-C-19-310446-R-APP 
202 | PUBLICUSER300151 | 9/3/2019 9:59:57 AM LAC-19-310019-R- APP 
203 | PUBLIGUSER245868 | 9/3/2019 9:59:58 AM 1A:6-19:310015-R-APP 
204 | PUBLICUSER280158 | 9201995058 AM | LAC-19-310207-R-APP 
205 | PUBLIGUSER289478 | 9/3/2019 9:59:58 AM TA.0-19-310230-R-APP 
206 | PUBLICUSER200024 | 9/3/2019 9:59:58 AM LAG-19-310085-R-APP 
207 | PUBLICUSER293733 | 913/2019 9:59:58 AM LAC-19-310161-R-APP 
208 | PUBLICUSER296064 | 9/3/2019 9:59:58 AM LA-6-19:310357-R-APP 
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209 | PUBLICUSER299682 9/3/2019 9:59:58 AM LA-C-19-310500-R-APP 
210 | PUBLICUSER299705 9/3/2019 9:59:58 AM LA-P-19-310267-R-APP 
211 | PUBLICUSER300580 9/3/2019 9:59:58 AM LA-C-19-310077-R-APP 
212 | PUBLICUSER242226 9/3/2019 9:59:59 AM LA-C-19-310317-R-APP 
213 | PUBLICUSER245485 9/3/2019 9:59:59 AM LA-C-19-310355-R-APP 
214 | PUBLICUSER288782 9/3/2019 9:59:59 AM LA-C-19-310522-R-APP 
215 | PUBLICUSER290635 9/3/2019 9:59:59 AM LA-C-19-310391-R-APP 
216 | PUBLICUSER290738 9/3/2019 9:59:59 AM LA-C-19-310458-R-APP, LA-C-19-310778-R-APP 
217. | PUBLICUSER291937 9/3/2019 9:59:59 AM LA-C-19-310197-R-APP 
218 | PUBLICUSER293288 9/3/2019 9:59:59 AM LA-C-19-310383-R-APP 
219 | PUBLICUSER294212 9/3/2019 9:59:59 AM LA-C-19-310541-R-APP 
220 | PUBLICUSER294262 9/3/2019 9:59:59 AM LA-C-19-310190-R-APP, LA-C-19-310465-R-APP 
221 | PUBLICUSER300061 9/3/2019 9:59:59 AM LA-0-19-310269-R-APP 
222 | PUBLICUSER301747 9/3/2019 9:59:59 AM LA-P-19-310577-R-APP 
223 | PUBLICUSER303050 9/3/2019 9:59:59 AM LA-C-19-310103-R-APP 
224 | PUBLICUSER303425 9/3/2019 9:59:59 AM LA-C-19-310220-R-APP 
225 | PUBLICUSER303577 9/3/2019 9:59:59 AM LA-C-19-310143-R-APP 
226 | PUBLICUSER304004 9/3/2019 9:59:59 AM LA-C-19-310187-R-APP 
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